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Cross-modal  congruency  benefits  for  com¬ 
bined  tactile  and  visual  signaling 

JAMES  L  MERLO 

United  States  Military  Academy,  West  Point 

AARON  R,  DULEY 
NASA  Ames  Research  Center 

PETER  A.  HANCOCK 

University  of  Centra!  Florida,  Orlando 

This  series  of  experiments  tested  the  assimilation  and  efficacy  of  tactile  messages  that  were 
created  based  on  five  common  military  arm  and  hand  signals.  We  compared  the  response  times 
and  accuracy  rates  for  these  tactile  representations  against  responses  to  equivalent  visual  repre¬ 
sentations  of  the  same  messages.  Experimentally  such  messages  were  displayed  in  either  tactile 
or  visual  forms  alone,  or  using  both  modalities  in  combination*  There  was  a  performance  benefit 
for  concurrent  message  presentations,  which  showed  superior  response  times  and  improved 
accuracy  rates  when  compared  with  individual  presentations  in  either  modality  alone.  Such 
improvement  was  due  largely  to  a  reduction  in  premotor  response  time.  These  improvements 
occurred  equally  in  military  and  nonmilitary  samples.  Potential  reasons  for  this  multimodal  fa¬ 
cilitation  are  discussed.  On  a  practical  level,  these  results  confirm  the  utility  of  tactile  messaging 
to  augment  visual  messaging,  especially  in  challenging  and  stressful  environments  where  visual 
messaging  is  not  feasible  or  effective. 


Humans  rdy  on  their  iiiultipte  sensory  systems  to 
continuously  integrate  the  etwironmental  stimuli 
around  them.  This  integration  allows  them  to  build 
their  perception  of  the  world  in  which  they  live. 
Although  each  sense  ahnie  is  remarkably  adept  at 
detection,  it  is  the  combination  and  integration  of 
these  disparate  .sensoiy  inputs  that  provide  the  rich 
tapestry  of  s])atial,  temporal,  and  object-related  infor¬ 
mation  on  which  buinarus  rely  to  sumve  and  thrive* 
Cross-modal  fusion  of  these  information  sources 
often  j^roves  more  beneficial  tlian  simply  increasing 
information  from  only  one  sensor\^  modality.  For  ex- 

i  i  * 

ample,  HilHs,  Ernst,  Banks,  and  Landy  (2002)  fountl 


that  when  combined,  the  value  ofmultiple  visual  cues 
(e*g*,  disparity  and  texture  gradients)  did  not  pro¬ 
duce  as  accurate  performance  as  when  both  visual 
and  tactile  cues  were  provided  in  an  object  properly 
discrimination  task*  Comparing  performance  within 
the  same  modality  with  combinations  of  two  or  more 
different  modalities  illustrates  that  information  loss 
can  occur  during  intramodal  presentations  that  does 
not  occur  with  the  fusion  across  different  modalities* 
In  the  .specific  case  of  tactile  and  visual  information 
there  seems  to  be  a  highly  efficient  integration  of  the 
two  sources  (Ernst  8c  Banks,  2002)*  Tins  integration 
is  especially  beneficial  when  the  cross-modal  cues 
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are  coiignictit  and  tnatdi  tiie  top-down  expectancies 
generated  by  past  experience. 

Previous  researcli  on  cross-modal  effects  has 
overwhelmingly  used  simple  forms  of  stimulation 
in  ex]3crinientatiom  For  example,  such  stimuli  are 
composed  of  the  illumination  of  a  simple  light  dis- 
play;  tactile  signals  are  represented  by  one  single 
point  stimuiation.  Much  may  be  learned  about  the 
basic  |>sycliopliysics  of  cross-modal  effects  tlirough 
lliese  laboratory-based  forms  of  sigtiaL  However, 
real-world  patterns  of  stimulation  are  often  com¬ 
plex  and  convey  important  meaning  beyond  rec¬ 
ognition  of  a  simple  sensor)^  change.  Unfortunately, 
very  little  is  known  about  cross- modal  integration 
effects  when  the  presented  stimuli  are  complex. 
Extrapolation  from  tlie  basic  understanding  of  the 
fundamental  psychophysics  to  the  actual  presenta¬ 
tion  of  real-w'orld  signals  is  neither  as  pristine  nor 
as  linear  as  is  often  expected.  Rich,  meaningful,  and 
information-laden  signals  are  not  always  treated  as 
linear  assemblies  of  simple  signals.  Thus,  although 
the  findings  from  extant  basic  research  establish¬ 
ing  cross-modal  advantages  {e,g,,  Spence,  Pavani, 
8c  Driver,  2004)  are  important,  sucli  advantages 
have  yet  to  he  fully  established  in  relation  to  real- 
world  |)erfonnance  tasks,  Tlierefore,  the  primary 
purpose  of  the  present  work  was  to  examine  such 
cross-modal  congruency  effects  in  circumstances 
using  direct,  meaningful,  real-world  signals.  In  these 
more  applied  settings,  if  the  cross-modal  advantage 
of  the  integration  of  visual  and  tactile  information 
is  confirmed,  it  could  improve  on  single-modality 
communication,  Suciv  an  advantage  would  be  es¬ 
pecially  evident  when  any  particular  sense  is  over¬ 
loaded  or  otherwise  degradecl  by  some  local  mask¬ 
ing  conditions.  For  example,  in  extreme  operational 
conditions,  sucli  as  militar>'  combat  or  firefighting, 
the  capacity  to  create  and  retain  some  form  of  redun¬ 
dancy  gain  is  not  merely  useful,  it  may  even  prove 
critical  to  survival  (1  lancock  Sc  Szalma,2oo8),  This 
pursuit  of  increased  communication  capacities  is 
important  because  missed  or  misinterpreted  signals 
or  messages  in  such  situations  can  have  catastrophic 
consequences  (Reason,  2008), 

Humans  not  only  rely  on  their  multiple  sensor)^ 
capacities  to  integrate  different  forms  of  stimuli,  they 
also  use  these  multiple  sources  to  aid  them  in  the  ini¬ 
tial  process  of  orientation  and  the  subsequent  focus 


of  their  attention  in  space  and  lime.  When  a  person 
directs  her  or  his  attention  toward  a  particular  loca¬ 
tion,  regardless  of  the  primary  modality  used  in  the 
process  of  detection,  the  other  modalities  are  often 
directed  toward  that  same  location.  Indeed,  there  is 
an  ongoing  debate  about  whether  the  orientation  of 
attention  is  a  multisensory  construction  (Spence  8c 
Driver,  2004)  or  an  overdominantly  visual  process 
(Posner,  Nissen,  8c  Klein,  1976),  Tliis  issue  can  be 
approaclied  from  a  neurophysiological  perspective. 
For  example,  Stein  and  Mereditli  (199.^)  showed  tliat 
bimodal  and  triinodal  neurons  have  a  stronger  cellu¬ 
lar  response  when  animals  arc  presented  witli  stimuli 
from  two  sensory  modalities  as  cornjiared  witli  stimu¬ 
lation  from  only  one  modality.  The  combinations  of 
two  different  seiisorj'  stimuli  significandy  enhance  the 
responses  of  neurons  in  the  superior  colliculus  com¬ 
pared  with  those  evoked  by  cidier  unimodal  stimulus 
alone-  This  observ^ation  supports  the  conclusion  that 
there  is  a  multisensory^  link  between  individual  supe¬ 
rior  colliculus  neurons  for  cross-modality  attention 
and  orientation  behaviors  (see  also  Meredith  8c  Stein, 
1996;  Wallace,  Meredith,  Sc  Stein,  1998). 

Recent  findings  also  reinforce  the  proposition 
that  multisensory'  processing  is  ]>ossible  for  unimodal 
neurons,  Allman  and  Meredith  (2007)  used  cellular 
recordings  to  measure  responses  of  neurons  in  the 
postlateral  lateral  suprasylvian  of  the  cat.  Altliough 
unimodal  visual  neurons  did  not  respond  when  pre¬ 
sented  only  auditory'  stimuli,  they  did  have  an  en¬ 
hanced  visual  response  with  concurrently  presented 
auditory^  stimuli.  This  finding  indicates  that  bimodal 
and  triinodal  neurons  are  not  the  exclusive  domain 
for  multimodal  processing,  ft  suggests  that  poten¬ 
tially,  there  is  a  subthreshold  multisensory  neuron 
that  contributes  to  the  processing  of  multimodal 
stimulation.  This  may  be  a  basis  for  the  finding  that 
behavioral  responses  to  bimodal  stimuli  are  faster  and 
more  accurate  tlian  for  unimodal  stimuli  (Teder-Sale- 
jar\'i,  Di  Russo,  McDonald,  &:  Hillyard,2005),  Prob¬ 
lematically,  multimodal  stimulation  in  the  real  world 
is  not  always  presented  or  received  in  a  congruent 
spatial  and  tem|)oral  maimer.  This  ambiguity  can  be 
resolved  by  overreliance  on  the  one  single  dominant 
system,  which  in  humans  is  expressed  in  the  visual 
modality  (Hancock,  2005),  However,  when  there  is 
a  strong  expectation  from  past  experience  tliat  real- 
workl  multisensory^  information  will  be  congruent 
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and  consistent,  but  it  proves  not  to  be  so,  the  result 
is  often  ] perceptual  error. 

As  we  have  noted,  to  date  the  exploration  of  cross- 
modal  attention  has  relied  mainly  on  simple  stimuli  to 
elicit  resjponses  (Spence  8c  Walton,  2005),  As  a  spe¬ 
cific  example,  Gray  and  Tan  (2002)  used  a  nimiher  of 
taclors  (vibrotactilc  actuators)  spanning  tl^e  lengtii  of 
die  )>articipant’s  ann  witSi  lights  mounted  on  die  indi- 
vklual  tactors.  Using  an  appropriate  iiitcrstimulus  in- 
ter\'al  and  tactor  spacing  (see  Geldard,  1982;  Geldard 
&  Sherrick,  1972;  Ilelson  8c  KJng,  1931),  Gray  and 
Tan  created  the  ilkision  of  movement,  either  up  or 
dow n  tlie  arm.  These  researchers  found  diat  response 
times  were  shorter  when  the  visual  target  was  offset 
in  the  same  direction  as  the  tactile  motion  (similar  to 
die  [iredietivc  abilities  one  has  to  know  die  location  of 
an  insect  when  it  runs  up  or  down  tlie  arm).  Reaction 
times  were  longer  wlieti  the  target  was  offset  in  the 
direction  opposite  to  the  tactile  moiioiL  This  further 
supports  tlie  contention  tliat  the  cross-modal  links 
hetw'een  vision  and  touch  are  updated  dynamically 
for  moving  objects  and  are  best  supported  perce]Ptu- 
ally  wlie[i  die  stimuli  are  congruent.  However,  it  also 
illustrates  again  that  extant  work  lias  used  only  very 
simple  patterns  of  stimulation. 

In  another  such  study,  Craig  (2006)  had  par¬ 
ticipants  judge  the  direction  ofajiparent  motion  by 
stimulating  two  locations  sequentially  on  a  partici¬ 
pant’s  finger  pad  using  vihrotactors.  Visual  trials  in¬ 
cluded  apparent  motion  induced  by  the  activation 
of  two  lights  sequentially  Some  trials  were  recorded 
wadi  both  visual  and  tactile  stlnmli  presented  together 
either  cemgruendy  or  incongniendy.  When  visual  mo¬ 
tion  was  jjresented  at  die  same  time  as  but  in  a  direc- 
don  opposite  to  tactile  motion,  accurac}^  in  judging  the 
direction  of  tactile  apparent  motion  was  substantially 
lower.  Tills  superior  perlonnaEice  during  congruent 
presentation  was  called  the  congruency  effect.  Suybel 
atul  Valakis  (2004),  who  used  visual  apparent  motion, 
conductetl  a  similar  ex|ieriment.  They  found  similar 
effects  for  Judgments  of  auditory  apparent  motion. 
Auditor)'  stimuli  have  also  been  show'ii  to  affect  the 
perceived  direction  of  tactile  apparent  motion  (see 
Soto-Faraco,  Spence,  8c  Kingstone,  2004a,  2004I)). 

Bensmaia,  Killebrew,  atul  Craig  (2006)  had  par¬ 
ticipants  make  discrimination  judgments  compar¬ 
ing  pairs  of  tactile  stimuli  with  drifting  sinusoids. 
On  some  of  the  trials  a  visual  drifting  sinusoid  was 


presented  simultaneously  witli  one  of  tlie  tw^o  tactile 
stirnulL  Tlie  visual  stimuli  served  as  a  distraction  that 
was  to  be  igiiored.  When  the  directions  of  drift  tor 
the  visual  and  tactile  grating  dis])lays  w'ere  congru¬ 
ent,  the  presence  of  the  visual  distractor  increased 
the  perceived  speed  of  the  tactile  grating.  When  the 
tw'o  stimuli  were  iiicongruent  {i.c.,  they  drifted  in 
opposite  directions),  the  effect  of  die  visual  distrac¬ 
tor  was  to  reduce  the  perceived  speed  of  the  tactile 
grating,  Altliough  all  these  experiments  widi  simple 
tasks  are  essential  for  understanding  the  psychologi¬ 
cal  phenomena  under  consideration,  the  extension 
of  these  findings  into  real-world  conditions  to  em¬ 
brace  more  applied  stimuli  is  essentially  unexplored. 
However,  with  advancements  in  tactile  display  tech¬ 
nology  and  innovative  signaling  techniques,  the  im¬ 
portance  of  evaluating  the  occurrence  and  jiower  of 
these  multimodal  enliancements  in  systems  capable 
of  assisting  actual  field  communicatioiis  is  now  both 
feasible  and  pragmatically  important.  Thus,  the  goal 
of  the  present  sequence  of  experiments  was  to  exam¬ 
ine  combinations  of  visual  and  tactile  communica¬ 
tions  of  real-world  operational  signals  in  order  to 
evaluate  whether  the  pattern  of  findings  from  simple 
forms  of  stimulation  persisted  in  real-world  circum¬ 
stances.  We  also  evaluated  whether  the  presentation 
of  inconsistent  multimodal  information  caused  any 
significant  change  in  response  cajiacity  using  these 
real-world  signals. 

EXPERIMENT  1 

METHOD 

Panidpants 

To  Ltivcstigatc  the  effeciiveness  ol  crass-modal  infor¬ 
mation  [presentation  of  complex,  real-world  comjilcx 
signals,  20  participants  (9  men,  11  women)  ranging 
in  age  from  18  to  48,  %vith  an  average  age  of  25  years, 
volunteered  to  participate  in  tlie  first  experiment. 
Eadi  jpariicipant  self-reported  tio  surgeries,  signifi¬ 
cant  scarring,  or  any  impediment  that  might  cause 
lack  of  feeling  in  the  abdomen  or  torso  area  where 
the  tactile  signals  vs'ere  presented.  Furthermore,  all 
[Participants  reported  iionnal  or  cor  reeled -to-nonnal 
vision.  Additionally  none  of  the  jpardcipants  had  any 
prior  experience  w'ith  either  the  visual  arm  and  hand 
signals  or  the  specific  form  of  tactile  signaling  used 
in  the  cxjpcriment. 
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Materials  and  apparatus 

The  vibrotactllc  actuators  (taciors)  used  in  the 
present  system  were  model  O2,  manufactured  by 
Engineering  Acoustics,  !nc*  Tliey  were  acoustic 
transducers  that  dis[>lace  200-  to  joo-Hz  sinusoidal 
vibrations  onto  the  skin.  Ilieir  ly^g  mass  was  suf¬ 
ficient  for  activating  tlie  skin’s  tactile  recejjtors.  The 
C2’5  contactor  was  7  nun,  w'iih  a  i-mm  gap  scfjarat- 
ing  it  from  the  tactor  aluTuirumi  housing.  The  O2  is 
a  tuned  device,  meaning  iliat  it  ojicrates  effectively 
only  w'llliiri  a  very'  small  fVec[ucucy^  range.  In  the  pres* 
ent  experiment,  tliis  ojjerationaf  range  was  centered 
on  250  1 1/,.  Tlie  tactile  dis[)lay  itself  was  a  hclt-likc 
device  w'itli  eight  vihrotactile  acttiators  embedded 
in  it.  Tliree  examples  oftite  presets  belt  system  are 
shown  in  Figure  1.  The  belt  itself  w  as  made  ofelas* 
tic  material  and  was  comjjosed  of  high-r|ua!ity  cloth 
similar  to  that  used  by  professional  cyclists.  When 
the  heh  is  stretched  around  tiie  l>ody  and  fastened, 
one  actuator  is  centered  over  the  umbilicus  and  one 
is  centered  over  the  spine.  The  other  six  actuators 
are  equally  sjjacetl  arountt  the  body,  three  on  each 
side,  fora  total  of  eight  (sec  also  Cholcw'iak,  BrilLfe 
Schw'ah,2004), 

n'he  tactors  are  activaietl  using  a  tactor  control 
unit.  Tills  is  a  com[>utcr-contro!!ed  driver  and  aru- 
])lificr  system  that  switciies  each  tactor  on  and  off  as 
commanded  by  the  associated  software.  Controller 
devices  are  also  sIiovvej  on  the  left  side  of  the  belts  in 
Figure  l.^'he  control  unit  w'eighs  1,2  Ih  indejiciidcnt 


of  its  powder  source  a  tK  I  isaj)pr(>ximalely  t  incli  thick. 
This  device  connects  to  a  power  source  w'ith  one 
cable  and  to  the  display  belt  with  the  other.  It  uses 
Bluetooth  technology'  ici  coininuriicate  with  the  com¬ 
puter-driven  interface.  We  created  tactile  messages 
based  on  five  standard  Army  and  Marine  Coq>s  arm 
and  band  signals  (Depariincnt  of  the  Army,  icjHy), 
Tlie  five  signals  chosen  tor  the  present  experiment 
w'cre  ‘VXttention,^’  lalt,”  ‘^Rally,”  “^^o^'e  Out,”  and 
‘‘Nuclear  liiological  Chemical  Event  (NBC),”  Tlie 
specific  tactile  representatinns  of  tliese  signals  were 
designed  in  a  coibhorative  effort  involving  a  consul* 
taut  group  of  subject  inaitcr  cxjierts  including  former 
U,S,  soldiers  and  Mariiies, 

We  create<I  five  short  video  clips  of  a  soldier  in 
uniform  perfonning  these  five  arm  and  hand  signals. 
These  video  clips  were  tlie  visual  stimuli  for  ilic  study, 
Carefui  editing  ensuretl  that  the  liming  of  the  arm 
and  hand  signals  matched  the  temporal  duration  and 
]>atierning  ofthe  tactile  presentations  (Figure  2).  \\e 
used  a  Samsung  Qi  Ultra  Mobile  com])uter  using  an 
Intel  Celeron  M  ULV  (900  MUz)  processor  with  a  y" 
WVGA  (800  X  48a)  licpiid  crj'stal  display  to  jiresem 
videos  of  the  soldier  jicrformitig  the  arm  anti  hand 
signals.  This  com])utcr  ran  a  custom  LahVIEW  (8,2; 
National  Instruments)  apjilication  that  presented  tlie 
com])arah!e  tactile  signals  via  Bluetooili  to  the  tac¬ 
tor  controller  hoard.  Tlie  ctunputer  also  captured  all 
partici|>unt  resjionses  via  mouse  input.  Participants 
wore  souud-tlamjiening  hcatlphoncs  wiili  a  reduc- 


FIGURE  1 .  Three  tactite  display  belt  assemblies  and  their  associated  controller  boKes 
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FIGURE  2,  Screen  shot  showing  what  the  participant  viewed  as  the  signals  were  presented. The  participant  mouse-dicked  on  the  apprO' 
priate  signal  name  after  each  presentation 


tion  rating  of  11.3  tlB  at  250  Hz.  This  precaution  was 
designed  to  mask  any  possible  effects  of  extraneous 
auditor)*  stimuli  produced  by  tactor  actuation.  Be¬ 
cause  this  issue  has  caused  controversv,  we  were 

it  ^ 

careful  to  control  for  this  potential  artifact  here  (cf. 
Broadbent,  1978;  Poulton,  1977)- 

Each  message  or  signal  was  displayed  in  one  of 
four  distinct  ways.  First  was  the  visual-only  condi¬ 
tion.  This  consisted  of  the  five  arm  and  hand  signals 
sltown  on  video.  Secontl,  tlic  tactile-only  j^resentaiion 
communicated  the  five  tactile  ec|iii\^ents  of  the  ann 
and  liand  signals. Tin rdjn  the  congruent-bo th  coridi- 
tiou  the  visual  fonn  and  the  tactile  fbnu  of  the  same 
signal  were  presented  siinultaneousiy.  Finally,  in  the 
incongnicnt-botli  condition  visual  and  tactile  signals 
were  jjresented  simultaneously,  but  the  visual  version 
did  not  malcli  tlie  tactile  version  of  the  signal. 

Design  and  procedure 

Participants  first  view'cd  a  computer-based  tutorial 
that  described  each  arm  and  hand  signal  individu¬ 
ally.  For  each  signal,  a  sliort  description  of  the  action 
was  presented.  Participants  then  viewed  a  video  of 
a  soldier  111  uniform  j^crforrnlug  the  signal,  followed 
by  a  direct  experience  of  its  tactile  ec|uivaleni.  Finally, 
the  jjarticipaiUs  were  able  to  experience  the  signals 
concurreiidy  (both  visual  and  tactile  congruent  rep¬ 
resentations  together).  Participants  were  allowed  £0 
repeat  this  presentation  (l.e.,  visual  only,  tactile  only, 
and  congruent  both)  as  many  times  as  they  desired. 


Once  the  participant  bad  completed  this  experi¬ 
ence,  a  validation  exercise  vvas  performed.  In  this 
exercise,  partici[>ant5  had  to  correedy  identify  each 
signal  twice  before  the  computer  would  prom[it  the 
ex jDerime liter  that  the  participant  was  ready  to  begin 
the  formal  experimental  [irocedurc. 

Each  ])articipant  performed  two  60-trial  blocks. 
Each  block  had  two  trials  of  each  signal  presented 
in  the  visiiabonly  condition  (lO  total  trials),  two  of 
each  signal  presented  in  the  tactile-only  condition 
(10  total  trials),  four  of  each  signal  in  the  coiigruent- 
botli  condition  (20  total  trials),  and  four  of  each  In  the 
incongruent-botb  condition  (20  total  trials).  These 
60  summed  trials  composed  one  block.  Eacli  partici¬ 
pant  performed  two  such  blocks.  Within  blocks,  tri¬ 
als  were  randomized  for  eacii  [iarticlj^ani.  The  entire 
experiment  took  just  less  than  an  hour  to  complete. 

RESULTS 

All  reportecl  analyses  were  conducted  using  an  al* 
pha  level  set  at  .05.  Results  were  analyzed  in  teniis 
of  the  speed  of  tlie  response  and  the  accuracy  of  the 
response  under  tlie  respective  performance  condi¬ 
tions.  Because  the  incongnieni-both  condition  raised 
a  numljcr  of  issues  in  analysis,  the  initial  comparisons 
are  derivetl  from  the  visual-only,  tactile-only, anti  ctm- 
gment-both  conditions  only.  An  analysis  of  variance 
(anova)  was  performed  on  the  mean  response  times 
across  the  tliree  described  experimental  conditions, 
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F\a,  37)  s  2.85,/;  <  .(>5,  =  .13Q,  P  =  0.653).  Siibse- 

qiicnt  painvisc  analysis  indicated  that  the  simultane¬ 
ously  presented  congment-both  signals  resulted  in 
significandy  sliorter  reS[)onse  limes  than  visual-only 
signals,  /(19)  =  -2/25, />  <  .04.  Analysis  also  Indicated 
tliat  congnjcnt-botli  signals  vt'ere  faster  tlian  those  for 
tactile-only,  /(19)  =  -3*98,/?  <  .ok  Finally,  the  visual- 
only  presentation  was  significantly  faster  than  tlie 
tactile-only  presentation,  /(19)  =  -2.16,^  <  .04.  These 
findings  are  illustrated  in  Figure  3. 

With  respect  to  res[>onse  accuracy,  a  marginal 
clilTereiicc  w'as  observed  between  the  visual-only 
atid  tactile-only  signals.  However,  this  just  failed  to 
reach  the  [ireset  level  of  significance,  ^(19)  =  2.00,  /j 
<  *06.  However,  there  was  a  significant  difference 
in  accuracy  wlicti  the  taciile-only  was  compared 
with  the  congnient-both  presentation  of  the  signals, 
^(19)  =  4,03,  /j  <  -Ot.  The  lo\ver  accuracy  rate  for  the 
tactile-only  appears  to  be  due  to  confusion  between 
the  tactile  signal  for  ^NliC”and  ^‘Halt.’NvIiicIi  have 
vcr\^  sinular  tactile  characteristics,  although  they  have 
a  very  low  level  of  siniilariiy  hi  the  visual  jjresenta- 
tions*  When  the  data  for  the  “NBC’’  tactile  signal 
were  reiiKwed,  though,  the  results  did  not  show  even 
the  marginal  difference  in  error  rate  between  visual- 
only  and  tactile-only  signals.  However,  such  removal 
did  not  affect  the  significant  main  effect  betw'een  the 
tactile  and  the  congruent-hoth  conditioru  After  this 
initial  com[>ar[son  between  visual-only,  tactile- only, 
and  congruent-both  cfmdittons  had  been  conducted, 
w'e  examined  comparisons  including  the  incongni- 
ent-botli  condition. 

Results  for  the  incongruent-both  condition 
were  analyzed  in  terms  of  the  sjieed  and  accuracy 
of  the  response.  During  liicongrueru-both  trials. 


the  participants  clmse  their  preference  for  tactile  or 
visual  presentation  (where  no  specific  instrucdons 
were  given  to  the  j>articipants  on  how'  to  deal  with 
this  conflict  of  signals).  In  some  cases,  neither  of 
the  presented  signals  was  selected  by  the  partici¬ 
pants,  therefore  die  response  was  coded  as  a  “re* 
S|)onse  not  matching  cither  presentation,”  with  9 
out  of  20  participants  making  tliis  type  of  error  at 
least  once.  Participants^  selections  were  examined 
during  incongrucnl-both  trials  lo  determine  which 
signal  modality  they  responded  to  most  frequently: 
There  was  an  overall  preference  for  choosing  the 
visual  ])  resell  tali  on  over  ttie  conflicting  tactile  signal. 
The  mean  response  time  for  the  incongruent-boih 
trials  is  also  shown  in  Figure  3. 

.'Mthoiigli  not  significantly  slower  llian  visual-only 
presentations,  /(19)  ==  ^0.150,  p  <  .88,  or  tactilc-oiily 
presentations, /( 19)  =  1.43, />  <  .17,  incongruent-oiily 
presentations  w'ere  significantly  slow'er  than  congru- 
eiil-oniy  presentations,  ^(19)  =  -3.778,  p  <  .01,  The 
responses  that  did  not  correspimd  with  either  mo¬ 
dality  of  the  presented  sigtial  were  not  considered  in 
this  comparison.  Tliis  meant  that  36  out  of  800  trials 
W'ere  omitted  from  that  analysis  and  analyzed  sepa¬ 
rately.  The  mean  rcs()onsc  times  of  these  trials  were 
computed,  and  Figure  3  show's  that  longer  response 
times  W'ere  associated  widi  these  respective  rcS}>o rises 
that  did  not  match  cither  of  tlie  incongruent  modal¬ 
ity  [iresentations.  Participants  reported  confusion 
on  not  being  told  w'luch  modality  took  jirecedence* 
Most  partici[>ants  slated  that  they  would  jiick  one 
modality  in  order  to  try  to  be  consistent.  However, 
others  re[iortccl  that  they  xvoiild  still  often  choose  the 
other  modality.  Thus,  responses  not  matching  either 
presentation  apparently  caused  a  longer  than  nor- 
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FIGURE  3.  Mean  response  times  (with  standard  error  bars)  by  modality  of  presentation.  Experiment  1 
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mal  decision-making  process,  as  represented  by  die 
longer  and  more  variable  response  times  here.  Tlie 
introduction  of  the  mcongruent-both  form  of  pre¬ 
sentation  proved  disconcerting  for  most  partici|>ants. 
The  majority  of  pardci[>ants  asked  for  clarification 
vvlien  the  first  or  second  incongruent-both  trial  was 
preseiiteci  during  the  randomised  setjuencc.  In  ati  ef¬ 
fort  to  support  experimental  fidelity,  tins  quesiioning 
by  die  participant  was  answered  widi  a  repeat  of  the 
instructions  that  they  were  to  follow. 

The  overall  high  accuracy  rate,  which  was  repre¬ 
sented  by  a  response  accuracy  of  more  than  8o%  in  all 
modalities  with  fewer  thati  lo  min  of  total  training,  is 
highly  encouraging  to  the  current  form  of  tactile  dis¬ 
play  for  real-world  applications.  The  accuracy  of  the 
messages  and  the  reported  inluitiveness  with  which 
they  were  received  confirmed  the  utility  of  the  pres- 
ently  selected  fonns  of  tactile  message.  Similarities 
between  various  tactile-only  and  visual-only  signals, 
w'hich  caused  potential  confusion  between  signals 
wlien  presented  in  one  modality  alone.,  was  essen¬ 
tially  eliminated  in  coiicurrent-both  presentations. 
The  confusion  and  subsequent  errors  created  in 
comjjarisons  of  the  tactile-only  signals  might  have 
been  exacerbated  by  tlie  presence  of  interj>olated 
incongriient-both  trials  {i.c.,  the  trials  in  which  tac¬ 
tile  and  visual  signal  did  not  match).  Overall,  the 
present  data  confirm  an  advantage  for  multimodal 
signal  presentation.  It  was  also  demonstrated  that  this 
advantage  was  not  due  to  any  trade-off  of  speed  for 
accuracy.  Tlie  next  logical  question  was  to  ascertain 
the  source  of  this  specific  processing  advantage.  This 
was  the  siihject  of  the  next  experimenl, 

EXPERIMENT  2 

METHOD 

Psrticipants 

Seventy-two  participants  (47  men,  25  women)  rang* 
iiig  ill  age  from  18  to  21,  with  an  average  age  of  18.5 
years,  volunteered  to  fiarticipate.  Of  these,  31  were 
from  the  Unwersity  of  Central  Florida,  and  the  re- 
maiiiing  41  were  from  tlie  IJ.S.  Military  Academy. 
The  [>rimar)'  difference  between  the  sam[>les  was 
that  the  latter  group  liad  prior  exjjerience  witli  the 
visual  form  of  the  presented  arm  and  hand  signals. 
However,  the  tactile  form  of  the  signals  was  new  to 
all  particij)ants.  As  in  Experiment  1,  each  participant 
had  no  impediment  that  could  cause  lack  of  feeling 
in  the  abdomen  or  torso  area  and  no  visual  im[>air- 


ments  that  would  obscure  their  capacity  to  respond 
to  the  visual  signals. 

Materiak  apparatus,  design,  and  procedure 

All  experimental  materials  and  apparatus  were  the 
same  as  those  described  in  Experiment  1.  The  ex¬ 
perimental  design  and  procedures  for  this  experi¬ 
ment  were  similar  to  those  of  the  first  experiment, 
with  a  small  number  of  important  differences.  First, 
all  incongruent-both  trials  were  removed  from  the 
procedure.  Therefore,  each  signal  was  presented  in 
one  of  only  three  ways:  the  visual-only  condition,  in 
which  the  participant  viewed  the  edited  video  pre¬ 
sentation  of  the  arm  and  hand  signals;  the  tactile- 
only  condition,  which  presented  the  comparable 
tactile  versions  of  the  arm  and  hand  signals;  and  the 
congruent-both  condition,  in  which  both  visual  and 
tactile  representations  of  t!ie  same  arm  and  hand  sig¬ 
nal  were  presented. 

Following  the  same  sequence  of  orientadoti  and 
practice  session  as  described  in  Experiment  i,  each 
particijjajit  completed  the  required  [jcrformance  tri¬ 
als.  TTie  participants  were  presented  each  of  the  five 
signals  eight  times  each.  They  were  presented  un¬ 
der  the  three  different  conditions— visual  only,  tactile 
only,  and  congruent  both— for  a  grand  total  of  120 
trials,  live  order  in  w  hicli  eacli  participant  perfonned 
the  sequence  of  trials  was  completely  randomized. 

The  primary  difference  in  the  present  experiment 
was  in  the  format  of  response.  Before  each  trial  began, 
tlie  participant  had  to  place  the  mouse  cursor  inside  a 
small  square  in  the  center  of  the  screen.  The  presen¬ 
tation  of  the  signal,  regardless  of  its  modality,  started 
the  tinier,  and  the  follawing  performance  responses 
were  collected:  the  initial  movement  of  tlie  mouse,  the 
latency  to  name  the  received  signal,  the  signal  named, 
and  tlie  accuracy  of  that  choice.  ITiis  format  permit¬ 
ted  us  to  parse  the  response  into  premotor  time  (the 
first  movement  of  the  mouse)  and  motor  time  (the 
time  to  place  the  cursor  in  the  response  box)  for  both 
correct  and  incorrect  resjionses.  These  responses 
were  subjected  to  analysis. 


Results  were  analyzed  in  terms  of  the  speed  of  the 
response  and  the  accuracy  of  the  response  under 
the  respective  conditions.  We  cotiducted  an  initial 
analysis  to  assess  any  potential  .sex  differences  across 
partidpants.  Howwer,  no  significant  influence  ol  this 
factor  was  found  on  any  of  the  measures  recorded, 


RESULTS 
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therefore,  t!ie  subsequent  analysis  was  collapsed 
across  the  sexes*  A  one-way  anova  was  performed 
on  the  mean  response  limes  across  the  tliree  experi¬ 
mental  conditions  of  visual-only,  tactile-only,  and 
congruent-both  presentations,  F{2^  213)  =  9*37,  p  < 
.01,  =  -.961,  P  =  i.oo.  Post  hoc  analysis  showed 

that  congment-botii  signals  resulted  in  a  significantly 
shorter  response  time  than  the  visual-only  condition, 
/(71)  =  3.15^^  <  .01*  Responses  to  the  congnieni-both 
signals  were  also  faster  than  tactile-only  responses, 
/(71)  =  10.29, /j  ^  Ad<litionally,tiie  visual-oniy  pre¬ 

sentation  of  die  signal  was  significandy  faster  than  the 
tactile-only  presentation,  ^(71)  =  -4.15,^  <  .01*  These 
results  are  illustrated  in  Figure  4.  The  data  were  exam¬ 
ined  for  effects  of  gender  in  the  different  modalities* 
A  general  linear  model  maxova  of  the  witliiii-subject 
variables  of  modality  (visual,  tactile,  and  congruent) 
and  belwecn-subject  variable  of  gender  (male  and 
female)  was  conducted  for  response  time  means, /T(2, 
36)  =  .642, p  <  .532,  n* ,  =  -034,  P  =  ■  I49i  blit  tliere  was  ' 
no  significant  difference  between  women  and  men  in 
any  of  the  modalities. 

Analysis  of  the  response  accuracy  data  showed  a 
significant  difference  in  accuracy  between  the  visual- 
only  and  tactile-only  conditions, /(71)  -  -7.10,J><  .01. 
This  difference  was  probably  due  to  the  extraordi¬ 
narily  liigli  accuracy  level  in  the  visual-oiJy  condition. 
Tliis  may  be  due  in  part  to  tlie  military  participants 
already  being  familiar  witli  and  having  some  previous 


training  in  the  visual  form  of  tlie  signals.  In  contrast, 
none  of  the  participants  had  any  prior  experience 
with  the  tactile  form  of  the  signals  presented  here. 
There  was  also  a  significant  difference  in  the  accuracy 
rate  when  responses  to  die  tactile-only  condition  were 
compared  w'ith  the  congruent-both  presentation, 
^(71)  =  7-47, *01,  with  the  congruent-botli  condi¬ 
tion  being  more  accurate.  Tlie  overall  lower  accuracy 
rate  for  tactile-only  signaling  is  again  attributed  to  the 
confusion  between  the  tactile  signal  for  ‘^NBC”  and 
“Halt.”  Analysis  without  the  “NBC"  tactile-only  sig¬ 
nal  data  removed  the  reportetl  significant  differences 
in  response  accuracy.  There  was  no  significant  differ¬ 
ence  between  responses  for  the  visual-only  condition 
and  the  combined-both  condition. 

A  one-way  axova  was  performed  on  the  mean 
response  times  for  the  premotor  element  (the  time 
that  elapsed  from  presentation  of  the  signal  to  the 
first  mov'emeiU  of  the  mouse)  across  the  three  experi- 
mentai  conditions.  This  analysis  showed  a  significant 
effect,  F{2,  213)  =  5.48,  p  <  .01,  =  .961,  p  =  1.00. 

Sul)set]uent  pairwise  com]>arisons  showed  dial  simul¬ 
taneously  presented  congruent-both  signals  resulted 
in  a  significandy  shorter  premotor  response  time  than 
visual-only  condition,  ^(71)  =  4*30,  <  .01.  Congru- 

ent-both  response  times  proved  significantly  shorter 
than  the  premotor  times  for  die  tactile-only  condition, 
/(71)  =  -2.9,^  <  .01.  Additionally,  premotor  response 
times  for  the  visual-only  presentation  were  signlfi- 
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candy  shorter  than  die  premotor  responses  for  tlie 
tactile-only  presentadon  oftlie  signal^ /(71)  =  -2,89,/? 
<  ,01,  These  results  are  also  illustrated  in  Figure  4*  In 
addition  to  die  preniotor  response  times,  the  present 
design  permitted  analysis  of  the  motor  response  times 
diemselves  that  represented  die  latency  to  move  to  die 
designated  response  location*  No  differences  were 
found  across  any  of  the  experimental  conditions  for 
motor  response  latency 

It  was  further  hypothesized  that  there  would  be 
some  differences  between  the  two  sample  groups  of 
students.  In  particular,  this  hypothesis  derived  from 
the  understanding  that  because  of  their  experience 
with  the  hand  signals  communicated, cadets  may  have 
greater  facility  in  response*  In  contrast,  the  university 
students  were  encountering  tliese  signals  for  the  first 
time*  To  a  degree,  any  such  initial  difference  ought  to 
have  been  mitigated  by  the  practice  given.  However, 
we  cliose  to  examine  this  liypotliesis  analytically  A  t 
test  did  detect  such  a  differetice,  w'hich  was  evident  in 
the  premotor  response  time  to  die  tactile-only  signals 
alone,  ^(70)  =  1*99,^  <  *01  (militar)^  cadets  =  785  ms 
vs,  university  students  ==  956  ms), 

DISCUSSION 

The  present  results  show  a  facilitation  in  process¬ 
ing  speed  when  two  consistent  signals  are  presented 
together  in  both  modalities.  This  outcome  exempli¬ 
fies  the  redundant  signals  effect  (see  Kinchla,  1974). 
Rather  than  a  statistical  advantage  derived  from  the 
race  between  the  activation  of  two  separated  streams 
of  incoming  information,  the  observed  facilitation 
appears  more  likely  to  be  due  to  a  process  of  co- 
activation  (Miller,  1982;  also  see  Rach,  Diederich, 
Sieenken,  8c  Colonius,  2010),  This  form  of  descrip¬ 
tive  explanation  is  supported  by  more  extensive  un¬ 
derstanding  of  the  underlying  neurophysiological 
processes  (see  Allman  Sc  Meredith,  2007),  Facilita¬ 
tion  miglu  have  derived  from  a  change  in  response 
decision  criterion,  but  the  effect  cannot  be  attrib¬ 
uted  to  a  trade-off  ofs|>eed  for  accuracy  because  tlie 
consistent-bolh  condition  was  significantly  more 
accurate  than  the  tactile-alone  presentation  and  as 
accurate  as  the  visual-alone  presentation  in  the  first 
experiment,  and  this  pattern  of  results  persisted  in 
the  second  experiment.  This  result  might  have  been 
affected  by  confusion  between  two  specific  forms  ol 


tactile  signal*  However,  even  if  tliis  were  the  case, 
the  consistent-hoth  condition  sliows  responses  that 
are  at  least  as  accurate  as  the  visual-only  and  tactile- 
only  conditions.  The  multimodal  advantage  is  not 
the  result  of  any  trade-off  ofsjjeed  for  accuracy  What 
emerges  is  a  genuine  advantage  in  performance  for 
the  multimodal  condition*  There  are  a  number  of 
potential  reasons  why  this  may  occur*  First,  we  can 
look  to  the  methods  of  the  first  experiment,  in  which 
incoiigruent  signals  (i,e,,  the  co-occurring  visual 
and  the  tactile  representations  were  those  of  differ¬ 
ent  signals)  were  embedded  in  the  design.  However, 
the  interpolation  of  the  inconsistent  signals  should 
have  had  a  deleterious  effect  on  performance,  not 
the  enhancement  we  observed*  If  the  interjection  of 
inconsistent  signals  had  an  inhibiting  effect,  then  the 
overall  enhancement  of  consistenl-both  condition 
may  be  even  greater  than  reported*  At  the  present, 
we  must  affirm  some  form  of  multimodal  advantage 
that  derives  from  the  facilitation  due  to  cross-rein¬ 
forcement  of  incoming  sensory  signals* 

In  the  course  of  these  experiments,  we  examined 
the  responses  of  both  men  and  women.  However, 
analysis  of  both  experiments  indicated  no  gender 
differences.  Because  the  tactile  signals  are  largely 
spatial  emulations  of  their  visual  counterpart,  this 
outcome  was  not  especially  surprising*  Gender  dif¬ 
ferences  found  in  spatial  cognition  tasks  often  are  due 
to  mental  rotation  differences,  and  little  or  no  mental 
rotation  seems  to  be  needed  for  tactile  signal  compari¬ 
son  or  interpretation  (Voyer,  Voyer,  8c  Biy^den,  1995)* 
Additionally,  some  researcliers  have  reported  a  de¬ 
cline  in  gender  differences  due  to  social  and  nurturing 
changes  in  today’s  more  gender-neutral  environment 
(Feingold,  1988)*  Furthermore,  other  factors  sudi  as 
athletic  [jartlcipatioii  and  specific  skills  development 
may  mask  or  negate  any  such  differences  (Hancock, 
Kane,  Scallen,  &  Albinson,  2002),  and  this  may  be 
especially  true  of  the  cadet  sample  in  Experiment  2. 
However,  it  should  be  acknowledged  that  the  sam¬ 
ple  size  in  the  present  experiments  was  small*  Thus, 
more  extended  cotnpaiisons  may  he  Justified  because 
demonstrated  differences  still  exist  for  many  spatial 
tasks  (see  Maccoby  Scjacklin,  1974),  and  these  trends 
are  well  substantiated  in  the  literature  (Masters  & 
Sanders,  1993)*  In  practical  terms,  though,  the  pres¬ 
ent  outcome  is  encouraging  because  it  suggests  that 
both  men  and  women  will  benefit  equally  from  any 
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tievelopeci  real-world  technologies  based  on  ihe  mul¬ 
timodal  acivantage  identified  here. 

To  return  to  the  issue  of  why  this  multimodal  ad¬ 
vantage  is  observed^  a  more  realistic  source  for  the 
enhancement  may  lie  in  the  neurophysiologic  archi¬ 
tectural  linkages  discussed  In  tlie  introduction  to  this 
article.  It  appears  that  cross-modal  reinforcement  has 
a  direct  effect  on  die  strength  of  synaptic  transmission 
that  is  experienced  early  in  the  stimulus  processing 
sequence*  Experiment  2  was  conducted  to  explore 
tins  possibility*  In  that  exjieriiiieiit^  the  response  was 
parsed  to  isolate  the  motor  output  component  of  the 
resjionse  sequence.  We  found  a  strong  confirmation 
of  the  multimodal  presentation  advantage  for  com¬ 
plex  stimuli  and  of  the  isolation  of  that  advantage 
into  the  early,  premotor  stages  of  response.  At  pres¬ 
ent,  it  is  uncertain  whetlier  the  primarj^  advantage 
is  to  be  found  in  the  perceptual  recognition  phase 
of  the  information-processing  sequence  or  in  the 
decision-making  and  response  formulation  element 
of  that  sequence.  However,  the  distinction  of  such  a 
difference  should  be  amenable  to  subsequent  identifi¬ 
cation.  From  the  present  results  it  appears  that  a  neu¬ 
rophysiological  argument  underlying  cross-modal 
stimulation  is  the  best  candidate  account  for  the  early 
advantage  offered  by  consistent  inuitimodal  signaling. 
Our  findings  further  affirm  that  the  multimodal  ad¬ 
vantage  demonstrated  with  very  simple  stimuli  does 
persist  when  the  signals  used  are  rich  in  Information 
and  meaning. 

Our  results  have  implications  beyond  the  expla¬ 
nation  of  the  physiological  dimensions  of  multimodal 
sensory  assimilation.  First,  there  Is  a  learning  dimen¬ 
sion  of  the  present  forms  of  signaling*  Whereas  the 
visual  forms  of  the  arm  and  hand  signals  have  been 
developed  and  used  by  the  military  over  an  extended 
period,  the  tactile  ei|iiivalents  were  developed  here 
only  for  the  purposes  of  the  present  study.  Never¬ 
theless,  all  participants  learned  these  forms  of  tactile 
signals  rapidly.  There  was  more  tlian  8o%  recognition 
of  tliese  signals  with  less  than  to  min  of  training.  This 
suggests  that  tactile  signaling  may  be  a  highly  effec¬ 
tive  fonn  of  coinmunicatlon  even  for  complex  signals. 
The  promise  of  tliis  facility  is  that  a  simple,  low-baud* 
rate  language  of  the  skin  may  eventually  be  developed 
for  wider  human  communication  capacities.  Such 
development  may  have  important  implications  for 
various  disabled  populations. 


The  absence  of  any  observed  gender  difference 
is  encouraging  for  the  subsequent  field  use  of  multi¬ 
modal  communication  capacities.  As  we  have  noted, 
such  signaling  can  thus  be  ubiquitously  adminis¬ 
tered  in  applied  operational  conditions.  The  find¬ 
ing  that  military  personnel  performed  better  in  the 
tactile  element  than  in  the  visual  form  with  which 
they  were  already  familiar  is  an  especially  intrigu¬ 
ing  result.  It  could  be  argued  that  because  of  their 
familiarity  with  the  visual  signals,  the  military  par¬ 
ticipants  exerted  a  greater  effort  to  learn  the  tactile 
signals  and  tlierefore  were  better  able  to  respond 
to  them.  However,  all  participants  had  an  equal  op¬ 
portunity  to  learn  all  the  signals  to  a  criterion  level, 
a  fact  that  militates  against  such  an  explanation.  It 
could  also  he  argued  that  lliis  reflected  a  sampling 
bias  because  military  cadets  arc  already  under  a 
greater  selection  pressure  when  initially  entering 
their  institution.  In  essence,  die  cadets  are  simply 
better  performers  on  an  absolute  level*  However,  if 
this  were  so  such  superiority  would  be  expected  to 
be  reflected  in  all  facets  of  performance.  Tlie  present 
data  do  not  su|>port  such  a  proposition.  At  present, 
tliere  is  no  evident  explanation  for  this  observ'ed  dif¬ 
ference,  and  it  is  possible  that  this  is  a  s])urious  find¬ 
ing.  However,  the  absolute  performance  difference 
was  large  and  consistent  and  therefore  is  unlikely  to 
be  B  ciiance  result. 

The  question  of  learning  complex  tactile  com¬ 
munication  signals,  especially  for  use  in  adverse  or 
unusual  circumstances,  is  an  important  future  issue 
in  both  practical  and  theoretical  realms*  The  tactile 
system  can  clearly  act  as  a  redundancy  gain  by  giving 
participants  mulU[>le  means  of  receiving  communica¬ 
tion.  In  practical  terms,  this  backup  line  of  commu¬ 
nication  might  vvell  assume  preeminence,  especially 
in  highly  adverse  conditions  (e.g.,  firefighting)  when 
obstmetions  to  vision  and  audition  make  other  forms 
of  communication  impossible.  It  may  also  he  a  criti¬ 
cal  alternative  avenue  of  information  presentation 
under  stress  (Hancock  &l  Warm,  1989).  Although 
the  current  findings  sliow  superior  performance  for 
the  recognition  of  tactile  arm  and  hand  signals  in  a 
multimodal  form  of  presentation,  many  challenges 
to  integrated  multimodal  signaling  in  the  real  world 
remain.  Stimulus-response  compatibility  (see  Proc¬ 
tor  &;  Vu,  tills  issue)  will  have  to  be  assessed  care¬ 
fully  to  maximi^fe  |)erformance  as  different  types  of 
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inputs  are  considered  for  use  with  combined  visual 
and  tactile  displays*  I  lowever,  when  people  are  faced 
with  extreme  challenges  and  traditional  sources  of 
visual  or  auditory'  information  are  diminished  or  de- 
gradetl  in  some  fashion,  tactile  stimulation  provi<!es 
an  important  augmented  communication  cliannel. 
The  fact  that  complex,  infoniiationally  rich  messages 
can  be  facilitated  by  such  multimodal  presentation  is 
itself  evidence  that  important  basic  research  findings 
can  be  extended  and  elalrorated  into  crucial  real* 
world  application* 

NOUS 

Address  correspondence  abotU  this  article  to  J.  L  Merlo, 
United  States  Military  Academy,  West  Point,  NY  (c- 
inail:  jariics.merlo@usiiia.edu). 
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